Network security is an important issue in modern computer networks. Technologies such as intrusion detection systems (IDSs) and firewalls are used extensively to protect computing resources on the network from unauthorized activities.
IDSs and firewalls are typically configured with a set of policies. These systems make decisions regarding controlling, monitoring and other related network administration activities based on the policies. In order to apply the policies, it is desirable to identify the type of network traffic. There are conventions that specify the mapping of application protocols to ports. Existing systems generally rely on this convention and use simple port characteristics to determine the types of network traffic going through the system. For example, the destination port for all HTTP traffic is typically port 80. A firewall configured to allow HTTP traffic identifies all network packets destined for port 80 as HTTP traffic, and allows them to pass through. A potential problem with this approach is that it does not take into consideration the application data of the packet. The application data of the packet may contain data for an application that is normally disallowed by the firewall; however, because the packet is destined for an allowed port, it typically bypasses firewall detection.
Using an allowed port to send disallowed traffic and avoid firewall detection (also referred to as “tunneling”) is easy to accomplish yet difficult to detect. Many applications such as instant messaging and peer-to-peer file sharing include built-in port scanning functions to detect ports allowed by the firewall, and use those ports to tunnel traffic that may be forbidden by the firewall. Tunneling also poses a threat to IDSs, which commonly rely on port mapping to determine the application of signatures. A packet destined for an allowed port containing disallowed traffic is typically either ignored, leading to no detection; or has incorrect signatures applied to it, leading to high rate of false negatives.
Problems also arise when services run on non-standard ports. The default behavior of most firewall systems is to disallow the traffic. For example, if a firewall is configured to allow HTTP traffic on port 80 only, traffic destined for a target server that runs its HTTP service on port 8080 is dropped and the user loses service. To provide users full service on non-standard ports requires opening more ports on the firewall, which increases the security risk. Services on non-standard ports are also problematic for IDSs. Since the traffic cannot be mapped to a specific protocol, the IDSs usually default to detecting everything or nothing at all. If the IDS attempts to detect everything, it tends to consume a lot of system resources (computing cycles, memory, etc) and increase the number of false positives. On the other hand, if the IDS detects nothing at all, any potential threat to the system would go undetected.
It would be desirable to have a system and method that could identify protocols without relying on the port mapping conventions. It would be useful if the technique could improve the accuracy of policy decisions without incurring significant system overhead. The present invention addresses such needs.